Keyframe-based recognition and localization during video-rate parallel tracking and mapping

Generating situational awareness by augmenting live imagery with collocated scene information has applications from gameplaying to military command and control. We propose a method of object recognition, reconstruction, and localization using triangulation of SIFT features from keyframe camera poses in a 3D map. The map and keyframe poses themselves are recovered at video-rate by bundle adjustment of FAST image features in the parallel tracking and mapping algorithm. Detected objects are automatically labelled on the user’s display using predefined annotations. Experimental results are given for laboratory scenes, and in more realistic applications.